The present invention relates environment display systems. More specifically, the present invention relates to using multiple video streams to increase the resolution of an immersive video systems.
As the processing power of microprocessors and the quality of graphics systems have increased, environment mapping systems have become feasible on consumer electronic systems. Environment mapping systems use computer graphics to display the surroundings or environment of a theoretical viewer. Ideally, a user of the environment mapping system can view the environment at any horizontal or vertical angle. FIG. 1 illustrates the construct used in conventional environment mapping systems. A viewer 105 (represented by an angle with a curve across the angle) is centered at the origin of a three dimensional space having X, Y, and Z coordinates. The environment of viewer 105 (i.e., what the viewer can see) is ideally represented by a sphere 110, which surrounds viewer 105. Generally, for ease of calculation, sphere 110 is defined with a radius of 1 and is centered at the origin of the three dimensional space. More specifically, the environment of viewer 105 is captured and then re-projected onto the inner surface of sphere 110. Viewer 105 has a view window 130 which defines the amount of sphere 110 viewer 105 can see at any given moment. View window 130 is typically displayed on a display unit for the user of the environment mapping system.
Conventional environment mapping systems include an environment capture system and an environment display system. The environment capture system creates an environment map which contains the necessary data to recreate the environment of viewer 105. The environment display system displays portions of the environment in view window 130 based on the field of view of the user of the environment display system. An environment display system is described in detail by Hashimoto et al., in co-pending U.S. patent application Ser. No. 09/505.442, entitled xe2x80x9cENVIRONMENT DISPLAY USING TEXTURE PROJECTIONS WITH POLYGONAL CURVED SURFACES.xe2x80x9d Typically, the environment capture system includes a camera system to capture the entire environment of viewer 105.
Computer graphic systems are generally not designed to process and display spherical surfaces. Thus, as illustrated in FIG. 2, texture mapping techniques are used to create a texture projection of the inner surface of sphere 110 onto polygonal surfaces of a regular solid (i.e., a platonic solid) having sides that are tangent to sphere 110. As illustrated in FIG. 2, a common texture projection is to use a cube 220 surrounding sphere 110. Specifically, the environment image on the inner surface of sphere 110 serves as a texture map which is texture mapped onto the inner surfaces of cube 220. A cube is typically used because most graphics systems are optimized to use rectangular displays and a cube provides six rectangular faces. Other regular solids (i.e., tetrahedrons, octahedrons, dodecahedrons, and icosahedrons) have non-rectangular faces. The faces of the cube can be concatenated together to form the environment map. During viewing, the portions of the environment map that correspond to view window 130 (FIG. 1 and FIG. 2) are displayed for viewer 105. Because, the environment map is linear, texture coordinates can be interpolated across the face of each cube based on the vertex coordinates of the faces during display.
Other texture projections can also be used. For example, cylindrical mapping, as illustrated in FIG. 3, can be used if view window 130 is limited to a visible range around the equator. Specifically, in FIG. 3, a texture projection in the shape of a cylinder 320 surrounds sphere 110. Portions of the environment image on the inner surface of sphere 110 serves as a texture map which is texture mapped onto the inner surfaces of cylinder 320. Often, cylinder 320 is approximated using a plurality of rectangular sides to simplify the texture mapping. FIG. 4 illustrates a texture projection ideally suited for environment mapping. Specifically, FIG. 4 shows a texture projection comprising a plurality of polygonal curved surfaces, such as polygonal curved surfaces 410, 420, and 430. The polygonal curved surfaces form a sphere 400 having the same radius as sphere 110 (FIG. 1). The environment image on the inner surface of sphere 110 serves as a texture map for the polygonal curved surfaces. Creation and use of polygonal curved surfaces in environment projection is described in co-pending U.S. patent application Ser. No. 09/505.442, entitled xe2x80x9cENVIRONMENT DISPLAY USING TEXTURE PROJECTIONS WITH POLYGONAL CURVED SURFACES.xe2x80x9d
An extension to environment mapping is generating and displaying immersive videos. Immersive video involves creating multiple environment maps, ideally at a rate of 30 frames a second, and displaying appropriate sections of the multiple environment maps for viewer 105, also ideally at a rate of 30 frames a second. Immersive videos are used to provide a dynamic environment rather than a single static environment as provided by a single environment map. Alternatively, immersive video techniques allow the location of viewer 105 to be moved. For example, an immersive video can be made to capture a flight in the Grand Canyon. The user of an immersive video display system would be able to take the flight and look out at the Grand Canyon at any angle.
Difficulties with immersive video are typically caused by the vast amount of data required to create a high resolution environment map and the large number of environment maps required for immersive video. Thus, most environment mapping systems use very large environment maps, i.e. 1024xc3x971024 or 2048xc3x972048. Conventional high-quality immersive video systems would also require such high resolution environment maps to create high-quality immersive videos. However, conventional video equipment are designed and built to fixed standards regarding image resolution, compression, and other features, which may not provide enough bandwidth for high-quality immersive videos. For example, standard NTSC video streams provides an equivalent resolution of 640 by 480 pixels. While this resolution is adequate for conventional videos, it is inadequate for high-quality immersive videos, which must include the entire environment of a viewer not just the portions being viewed. Hence, there is a need for a method and system of displaying immersive videos while satisfying the constraints of conventional video equipment.
Accordingly, the present invention provides an immersive video system which utilizes multiple video streams to display high resolution immersive videos using conventional video equipment. In one embodiment of the present invention, an immersive video system for displaying a view window includes a video source, a video decoder coupled to the video source, and an immersive video decoder coupled to the video decoder. The video source is configured to produce a plurality of video streams. The video decoder is configured to decode an active video stream selected by the immersive video decoder from the plurality of video streams. The video streams contain environment data for the environment. Each video stream overlaps at least one other video stream so that the active video stream can be switched between the plurality of video streams without discontinuities. In some embodiments of the present invention, each video stream overlaps at least two other video streams. Generally the amount of overlap is approximately the size of the view window. Additionally, some embodiments of the present invention include view control interfaces to position the view window.
The video source in many embodiments of the present invention includes a video reader and a video storage medium. For example, a specific embodiment of the present invention uses DVD as the video storage medium. However, other embodiments may use other video storage mediums, such as a hard disk or solid state memories. The video storage medium contains multiple video streams each having a viewable range from a reference point. Generally, the viewable range of the video streams overlap. For example, in one embodiment of the present invention, the video storage medium stores four video streams. The viewable range of the first video stream overlaps the viewable range of the second and fourth video stream. Similarly, the viewable range of the third video stream also overlaps the viewable range of the second and fourth video streams.
The present invention will be more fully understood in view of the following description and drawings.